
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 13-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



10/692,999 



10/23/2003 



Robert R. Rice 



7590 



09/11/2006 



Connie M. Thousand 

Northrop Grumman Space & Mission Systems Corp. 
One Space Park, Bldg. El/2041 
Redondo Beach, CA 90278 



000404-804 



9334 



EXAMINER 



ALLISON. ANDRAES 



ART UNIT 



PAPER NUMBER 



2631 



DATE MAILED: 09/11/2006 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



OffiCG Ar*tinn Ssammarx/ 

K/ffiv^ /ll#lll/fl wU/flfffCIf Jf 


Application No. 

10/692,999 


Applicant(s) 

RICE ETAL. 


Examiner 

Andrae S. Allison 


Art Unit 

2631 





_ The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1)^ Responsive to communication(s) filed on October 23. 2003 . 
2a)D This action is FINAL. 2b)K This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-11 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) H Claim(s) £jl is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) KI The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 23 October 2003 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) Kl Notice of References Cited (PTO-892) 4) Q Interview Summary (PTO-413) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) p aP er No(s)/Mail Date. . 

3) Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-152) 
Paper No(s)/Mail Date October 23. 2003: . f\# ftM V<> 100$ 6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 7-05) 



Office Action Summary 



Part of Paper No./Mail Date 20060829 



Application/Control Number: 10/692,999 



Page 2 



Art Unit: 2631 

DETAILED ACTION 
Specification 

1 . The disclosure is objected to because of the following informalities: The phrase 
"sport of interest" should be read "spot of interest" because the word "sport" should be 
replaced with "spot". 

2. The disclosure is objected to because it contains an embedded hyperlink and/or 
other form of browser-executable code on page 6, [p][16]. Applicant is required to delete 
the embedded hyperlink and/or other form of browser-executable code. See MPEP § 
608.01. 

Appropriate correction is required. 



Claim Rejections - 35 USC § 103 

3. Claims 1-3, 5, 6, 8, 9 and 11 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Black et al (US Patent No.: 5,774, 591) in view of Pantic (NPL 
document titled "Expert system for automatic analysis of facial expression") further in 
view of Vining et al (US Patent No.: 6,785,410). 

As to claim 1 , Black discloses a method for sensing selected emotions (e.g. 
happiness; column 10, line 64) in a human subject (human face; column 4, line 26). 
Black teaches generating an image of substantially all of the face of a human subject 
(40, Fig 2). Black also teaches processing the image to identify movements (determine 
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image motion, column 7, line 34) in selected critical areas of the face (e.g. eye regions; 
column 7, line 35). Black also teaches comparing the identified movements in the 
selected critical areas with a database (temporal models stored on memory 28, see 
Fig1) that associates movements in selected critical areas with specific emotional (e.g. 
anger expression can be characterized by an initial flattening of the lip followed by a 
long downward curvature, followed by the mouth going back to a relax position; column 
20, lines 36-40) and physical conditions. 

However, Black does not teach associating movement in critical areas with 
physical conditions. Pantic teaches a method of analyzing non-verbal communication 
(page 881 , [p][3], lines 4-5). Pantic also teaches associating movement in critical areas 
(specific facial actions; page 881, [p][3], line 10) with physical conditions (physiological 
reaction, page 881 , [p][3], line 13). Therefore it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to have added the method of 
analyzing non-verbal communication of Pantic to the method of recognizing facial 
expression of Black for vision based gesture analysis which includes face action 
recognition and emotional classification of facial expressions (page 903, [p][3], lines 2- 
5). 

Note the discussion above, both Black and Pantic does not teach generating a 
report of the emotional and physical condition of the subject. Vining teaches a method 
of reporting findings (column 2, line 11-12) of an expert analysis of image data that 
includes the step of generating a report (column 3, line 37-38) of the emotional and 
physical condition of the subject. Therefore it would have been obvious to one of 
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ordinary skill in the art at the time the invention was made to have added the method of 
reporting findings of Vining to the method of recognizing facial expression of Black as 
modified by Pantic for creating a report from a database of expert findings so that "data 
mining and other analysis may be conducted" (column 3, lines 39-46). 

As to claim 2, Black teaches wherein the processing step comprises 
inputting a two-dimensional frame of the image (column 7, line 47). Black teaches 
scanning the image to locate the subject's face and determine it's relative position and 
extent (column 7, line 59). Black also teaches scanning the facial part of the image to 
detect the selected critical areas (column 7, line 59). Black further teaches repeating the 
preceding steps for a sequence of image frames (column 8, lines 41-45). Additionally, 
Black teaches recording frame-to-frame changes (recover motion parameters; column 
8, line 56) in critical areas of interest. Black further teaches recording frame-to-frame 
changes in critical area positions for purposes of tracking the positions while permitting 
limited movement of the subject (column 8, lines 59-60). 

As to claim 3, Black teaches a method wherein the step of recording frame-to- 
frame changes in critical areas of interest includes recording changes in spot area (see 
column 9, line 33-43 where motion of spot area such as brow regions 45, see Fig 2, are 
recovered and stored in memory 20). 

As to claim 5, note the discussion above, Pantic teaches wherein the comparing 
step makes use of a database that uses the facial action coding system (FACS) (page 
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901, [p][1] lines 6-7). 
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As to claim 6, Black teaches an apparatus for sensing selected emotions (facial 
gesture and recognition system; column 6, line 68 and column 7, line 1) in a human 
subject. Black also teaches optical imaging device (see column 7, line 44 where image 
acquisition system 8, see Fig 1, includes a camera) for generating an image of 
substantially all of the face of a human subject. Black further teaches an image 
processing module (16, see Fig 1) for processing the image to identify movements in 
selected critical areas of the face; a database (temporal models are stored in memory 
28) associates groups of facial movements with specific emotional (e.g. anger 
expression can be characterized by an initial flattening of the lip followed by a long 
downward curvature, followed by the mouth going back to a relax position; column 20, 
lines 36-40) and physical conditions of the subject; 

However, Black does not teach associating movement in critical areas with 
physical conditions. Pantic teaches a system for analyzing non-verbal communication 
(page 881 , [p][3], lines 4-5). Pantic also teaches associating movement in critical areas 
(specific facial actions; page 881, [p][3], line 10) with physical conditions (physiological 
reaction, page 881 , [p][3], line 13). Therefore it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to have added the system of 
analyzing non-verbal communication of Pantic to the facial gesture and gesture 
recognition system of Black for vision based gesture analysis which includes face action 
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recognition and emotional classification of facial expressions (page 903, [p][3], lines 2- 
5). 

Black further teaches a database analysis module (26, feature and gesture 
recognition detector; see Fig 1) for comparing the identified movements in the selected 
critical areas with the database. 

Note the discussion both Black and Pantic does not teach a report generator, for 
generating a report of the emotional and physical condition of the subject. Vining 
teaches a system for reporting findings (column 2, line 11-12) of an expert analysis of 
image data that includes a report generator (10, reporting system, see Fig 2) for 
generating a report of the emotional and physical condition of the subject. Therefore it 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to have added the system for reporting findings of Vining to the recognizing facial 
expression system of Black as modified by Pantic for creating a report from a database 
of expert findings so that "data mining and other analysis may be conducted" (column 3, 
lines 39-46). 

As to claim 8, Black teaches wherein the image processing modules comprises: 
the means for inputting a two-dimensional frame of the image is image acquisition 
system 8, see Fig 1 . 

the means for scanning the image to locate the subject's face and determine it's relative 
position and extent is image segmentation 12, see Fig1 . 

the means for scanning the facial part of the image to detect the critical areas of interest 
is image segmentation 12, see Fig1. 
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the means for repeating the preceding steps for a sequence of image frames is motion 
estimation system 16, see Fig1. 

the means for recording frame-to-frame changes in the critical areas of interest is 
motion estimation system 16, see Fig1. 

the means for recording frame-to-frame changes in critical area positions, for purposes 
of tracking the positions while permitting limited movement of the subject is region 
tracking system, 18, see Fig 1. 

As to claim 9, Black teaches an apparatus wherein the means for recording 
frame-to-frame changes in the critical areas includes means for recording changes in 
area (see column 9, line 33-43 where motion of spot area such as brow regions 45, see 
Fig 2, are recovered and stored in memory 20). 

As to claim 1 1 , note the discussion above, Pantic teaches wherein the database uses 
the facial action coding system (FACS) (page 901 , [p][1] lines 6-7). 

4. Claims 4 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Black et al (US Patent No.: 5,774, 591) in view of Pantic (NPL document titled "Expert 
system for automatic analysis of facial expression") further in view of Vining et al (US 
Patent No.: 6,785,410) further in view of Lee et al (US Patent No. 7,095,901). 



As to claim 4, neither Black, Pantic nor Vining teach a method wherein the step 
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of recording frame-to-frame changes in critical areas of interest includes recording 
changes in axial distance to facilitate detection of axial pulsing movements. Lee teaches 
a method of obtaining iris images (column 4, line 61) including the step of recording 
changes in axial distance (measuring the distance between a user's face and a camera; 
column 8, line 10-11) to facilitate detection of axial pulsing movements. Therefore it 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to have added the method of obtaining iris images of Lee to the method of 
recognizing facial expression of Black as modified by Pantic and Vining for measuring 
the distance between a user and a "camera more accurately, more conveniently and 
more quickly and to focus the camera automatically" (column 4, lines 2-4). 

As to claim 10, neither Black, Pantic nor Vining teach an apparatus wherein the 
optical imaging device includes means for measuring axial distance to a critical area of 
the face; and the means for recording frame-to-frame changes in critical area positions 
includes means for recording changes in axial distance, to facilitate detection of axial 
pulsing movements in a critical area. Lee teaches an apparatus for obtaining iris 
images (column 4, line 61) including means for measuring axial distance to a critical 
area of the face (703, distance measurer; Fig 12) and the means for recording frame-to- 
frame changes in critical area positions includes the means for recording changes in 
axial distance (703, distance measurer; Fig 12) , to facilitate detection of axial pulsing 
movements in a critical area. Therefore it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to have added the apparatus for 
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obtaining iris images of Lee to the facial gesture and gesture recognition system of 
Black as modified by Pantic and Vining for measuring the distance between a user and 
a "camera more accurately, more conveniently and more quickly and to focus the 
camera automatically" (column 4, lines 2-4). 



5. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Black et 
al (US Patent No.: 5,774, 591) in view of Pantic (NPL document titled "Expert system for 
automatic analysis of facial expression") further in view of Vining et al (US Patent No.: 
6,785,410) further in view of Yamamoto (US Patent No.: 6,549,664). 

As to claim 7, note the discussion, neither Black, Pantic nor Vining teach an 
apparatus wherein the optical imaging device comprises a charged-coupled device 
(CCD) camera producing a two-dimensional image. Yamamoto teaches a face-image 
processing apparatus (column 3, line 25) that comprises a charged-coupled device 
(CCD) camera (see Fig. 2) producing a two-dimensional image. Therefore it would have 
been obvious to one of ordinary skill in the art at the time the invention was made to 
have substituted the optical image device of Black as modified by Pantic and Vining with 
the face-image processing apparatus of Yamamoto for photographing of a person's face 
used for extracting a certain characteristic such as an eye or nostril (column 3, lines 29- 
35). 
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Conclusion 

The prior art made part of the record and not relied upon is considered pertinent 
to applicant's disclosure. 

Endrikhovski et al (US Patent No.: 7,003,139) is cited to teach a method for 
determining affective information for at lease one image. 

Tian et al (US Patent No.: 6,459,806) is cited to teach a method for automatic 
detection of neutral expression in digital images and video. 

Black et al (NPL Document "Recognizing Facial Expressions in Image Sequence 
Using Local Parameterized Models of Image Motion") is cited to teach a method of 
recovering and recognizing rigid and articulated motion of human faces. 

Yaser et al (NPL Document "Recognizing Human Facial Expressions From Long 
Image Sequence Using Optical Flow") is cited to teach an approach to the analysis and 
representation of facial dynamics for facial expression recognition an from image 
sequence. 

Inquiries 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Andrae S. Allison whose telephone number is (571 ) 
270-1052. The examiner can normally be reached on Monday-Friday, 8:00 am - 5:00 
pm, EST. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Chanh D. Nguyen can be reached on (571) 272-7772. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Andrae Allison , « 



August 31 , 2006 





